Display control method, information processing apparatus and non-transitory computer readable medium

ABSTRACT

An information processing apparatus includes a display controller. The display controller acquires positional information, which indicates a position where a mobile display device is being used, from the display device; acquires first information, which is different from the positional information and is associated with the positional information, from a device different from the display device; and differentiates a two-dimensional image, which represents an object defined three-dimensionally from one viewpoint direction and is displayed on the display device, according to the acquired first information, and displays the two-dimensional image as a moving image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of the U.S. patentapplication Ser. No. 15/943,713, filed on Apr. 3, 2018, now allowed,which claims priority under 35 USC 119 from Japanese Patent ApplicationNo. 2017-161182 filed Aug. 24, 2017.

BACKGROUND (i) Technical Field

The present invention relates to an information processing apparatus.

(ii) Related Art

Techniques for displaying an object three-dimensionally include a methodusing a single display screen, a method using plural display screensarranged three-dimensionally, a method using a three-dimensionaldisplay, and the like.

SUMMARY

According to a first aspect of the invention, there is provided adisplay control method including following steps. Acquiring positionalinformation, which indicates a position where a display device is beingused, from the display device, the display device is a mobile displaydevice. Acquiring first information, which is different from thepositional information and is associated with the positionalinformation, from a device different from the display device.Differentiating a two-dimensional image, which represents an objectdefined three-dimensionally from one viewpoint direction and isdisplayed on the display device, according to the acquired firstinformation, and displaying the two-dimensional image as a moving image.

According to a second aspect of the invention, in the display controlmethod according to the first aspect, an image of a virtual creaturedisplayed on the display device, a form of which is imitated by theobject, changes in association with at least the first information andthe positional information.

According to a third aspect of the invention, in the display controlmethod according to the second aspect, an image of a virtual creaturedifferent from the image of the virtual creature is additionallydisplayed on the display device in association with a change of thefirst information and the positional information.

According to a fourth aspect of the invention, in the display controlmethod according to the second aspect, the number of images of virtualcreatures displayed on the display device, forms of which are imitatedby the object, is different in association with the first informationand the positional information.

According to a fifth aspect of the invention, in the display controlmethod according to the first aspect, when the first information isinformation regarding weather and an abnormality of weather is notifiedthrough the information regarding weather, the two-dimensional image isdisplayed in a mode according to a content of the abnormality.

According to a sixth aspect of the invention, in the display controlmethod according to the fifth aspect, when the abnormality of weather isnotified through the first information, warning information is displayedon the display device.

According to a seventh aspect of the invention, in the display controlmethod according to the first aspect, when the first information isinformation regarding weather, a magnitude of change in display of thetwo-dimensional image is changed according to the information regardingweather.

According to an eighth aspect of the invention, in the display controlmethod according to the first aspect, the two-dimensional imageincreases on the display device and then decreases according to thefirst information.

According to a ninth aspect of the invention, there is provided aninformation processing apparatus including a display controller. Thedisplay controller acquires positional information, which indicates aposition where a mobile display device is being used, from the displaydevice. The display controller acquires first information, which isdifferent from the positional information and is associated with thepositional information, from a device different from the display device.The display controller differentiates a two-dimensional image, whichrepresents an object defined three-dimensionally from one viewpointdirection and is displayed on the display device, according to theacquired first information, and displays the two-dimensional image as amoving image.

According to a tenth aspect of the invention, in the informationprocessing apparatus according to the ninth aspect, an image of avirtual creature displayed on the display device, a form of which isimitated by the object, changes in association with at least the firstinformation and the positional information.

According to an eleventh aspect of the invention, in the informationprocessing apparatus according to the tenth aspect, an image of avirtual creature different from the image of the virtual creature isadditionally displayed on the display device in association with achange of the first information and the positional information.

According to a twelfth aspect of the invention, in the informationprocessing apparatus according to the tenth aspect, the number of imagesof virtual creatures displayed on the display device, forms of which areimitated by the object, is different in association with the firstinformation and the positional information.

According to a thirteenth aspect of the invention, in the informationprocessing apparatus according to the ninth aspect, when the firstinformation is information regarding weather and an abnormality ofweather is notified through the information regarding weather, thedisplay controller displays the two-dimensional image in a modeaccording to a content of the abnormality.

According to a fourteenth aspect of the invention, in the informationprocessing apparatus according to the thirteenth aspect, when theabnormality of weather is notified through the first information, thedisplay controller displays warning information on the display device.

According to a fifteenth aspect of the invention, in the informationprocessing apparatus according to the ninth aspect, when the firstinformation is information regarding weather, the display controllerchanges a magnitude of change in display of the two-dimensional imageaccording to the information regarding weather.

According to a sixteenth aspect of the invention, in the informationprocessing apparatus according to the ninth aspect, the displaycontroller increases and then decreases the two-dimensional image on thedisplay device according to the first information.

According to a seventeenth aspect of the invention, there is provided anon-transitory computer readable medium storing a program causing acomputer to: acquire positional information, which indicates a positionwhere a mobile display device is being used, from the display device;acquire first information, which is different from the positionalinformation and is associated with the positional information, from adevice different from the display device; and differentiate atwo-dimensional image, which represents an object definedthree-dimensionally from one viewpoint direction and is displayed on thedisplay device, according to the acquired first information, and displaythe two-dimensional image as a moving image.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the present invention will be described indetail based on the following figures, wherein:

FIG. 1 illustrates an appearance example of an information processingapparatus according to a first exemplary embodiment;

FIG. 2 illustrates an arrangement example of planar sensors;

FIG. 3 illustrates an arrangement example of other sensors;

FIG. 4 illustrates a hardware configuration example of the informationprocessing apparatus;

FIG. 5 illustrates a functional configuration example of a controlleraccording to the first exemplary embodiment;

FIG. 6 illustrates an example of a three-dimensional object displayed ona display of the information processing apparatus;

FIG. 7 illustrates changes that are made to a character displayed as thethree-dimensional object upon acceleration (collision) being applied toa top face of the information processing apparatus;

FIG. 8 illustrates changes that are made to the character displayed asthe three-dimensional object if acceleration (collision) is continuouslyapplied to the top face of the information processing apparatus;

FIG. 9 illustrates changes that are made to a structure displayed as thethree-dimensional object if acceleration (collision) is continuouslyapplied to the top face of the information processing apparatus;

FIG. 10 illustrates changes that are made to the character displayed asthe three-dimensional object upon pressure being applied to the head ofthe character;

FIG. 11 illustrates changes that are made to the character displayed asthe three-dimensional object if pressure is continuously applied by afingertip to the head of the character;

FIG. 12 illustrates changes that are made to the character displayed asthe three-dimensional object upon pressure being applied to the chest ofthe character;

FIG. 13 illustrates changes that are made to the structure displayed asthe three-dimensional object upon distortion being applied to theinformation processing apparatus;

FIG. 14 illustrates changes that are made to the character displayed asthe three-dimensional object upon a temperature change being detected bytemperature detection sensors;

FIG. 15 illustrates a change that is made to an ice cube displayed asthe three-dimensional object upon a temperature change being detected bythe temperature detection sensors;

FIG. 16 illustrates a change that is made to the character displayed asthe three-dimensional object upon a humidity change being detected byhumidity detection sensors;

FIG. 17 illustrates an example in which acceleration detection sensorsare arranged in M rows and N columns in the plane of the display;

FIG. 18 illustrates a change in the number of times of detection ofacceleration (collision) from a specific position in a case in which theacceleration detection sensors are arranged in 4 rows and 4 columns;

FIG. 19 illustrates an appearance example of an information processingapparatus including displays on four faces, which are a front face, sidefaces, and a back face;

FIG. 20 illustrates an appearance example of an information processingapparatus according to a second exemplary embodiment;

FIG. 21 illustrates a hardware configuration example of the informationprocessing apparatus;

FIG. 22 illustrates a functional configuration example of a controlleraccording to the second exemplary embodiment;

FIGS. 23A to 23E illustrate examples of a bent position determined by abent position determiner;

FIG. 24 illustrates an example in which an image of thethree-dimensional object is edited by using information on thedetermined bent position;

FIG. 25 illustrates another example in which an image of thethree-dimensional object is edited by using information on thedetermined bent position;

FIG. 26 illustrates an example in which a deformation operation of theinformation processing apparatus is used to control a display operationof a display device whose display is controlled by another informationprocessing apparatus;

FIG. 27 illustrates another example in which a deformation operation ofthe information processing apparatus is used to control a displayoperation of a display device whose display is controlled by anotherinformation processing apparatus;

FIG. 28 illustrates an example of a three-dimensional object displayedby a three-dimensional display;

FIG. 29 illustrates a state in which a change is made to a displayedthree-dimensional object in response to a specific movement of a user;

FIG. 30 illustrates an example in which a change is made to athree-dimensional object that is projected onto a wall or a floor; and

FIG. 31 illustrates an example of a correspondence table representing acorrespondence relationship between objects as change targets incombination with numbers of dimensions of sensor values and changedimages.

DETAILED DESCRIPTION

Now, exemplary embodiments of the present invention will be described indetail below with reference to the attached drawings.

Terminology

In the exemplary embodiments described below, “to display an objectthree-dimensionally” means to display the object in a manner includingdepth information by using a single display screen, plural displayscreens arranged three-dimensionally, a so-called three-dimensionaldisplay, or the like.

A method using a single display screen includes a case of displaying atwo-dimensional image (image including information regardingperspective) obtained by actually taking a picture of an object that ispresent in a real space, a case of displaying a two-dimensional imagerepresenting an object defined in a three-dimensional space from oneperspective, and the like. Here, examples of the object defined in athree-dimensional space include a character in a virtual space, athree-dimensional image reconstructed from tomographic images, and thelike.

In addition, the method using plural display screens arrangedthree-dimensionally includes a case of displaying on the respectivedisplay screens, plural two-dimensional images obtained by observing anobject defined in a three-dimensional space from plural perspectives(corresponding to arrangement positions of the display screens).

In addition, the method using a three-dimensional display includes amethod requiring an observer to wear glasses having special opticalcharacteristics, a method requiring no such special glasses, a methodrequiring an observer to wear a head-mounted display on the head, andthe like. Examples of the method requiring no glasses include a methodusing the phenomenon that air at the focal point of condensed laserbeams changes into plasma and emits light.

However, as long as the object is represented three-dimensionally,information corresponding to the object does not have to necessarilyinclude internal information (voxel defining volume data), and may be,for example, a collection of multi faces (polygon mesh).

In the exemplary embodiments described below, “to display an objecttwo-dimensionally” means to display the object in a manner notcontaining depth information by using a single display screen, pluraldisplay screens arranged three-dimensionally, a so-calledthree-dimensional display, or the like.

In the exemplary embodiments, an object defined in a three-dimensionalspace is referred to as a three-dimensional object, and an objectdefined in a two-dimensional space is referred to as a two-dimensionalobject.

An object may alternatively be displayed three-dimensionally bydisplaying two-dimensional objects corresponding to plural displayapparatuses arranged three-dimensionally.

The object in the exemplary embodiments may be displayed as a stillimage or a moving image.

First Exemplary Embodiment Apparatus Configuration

FIG. 1 illustrates an appearance example of an information processingapparatus 1 according to a first exemplary embodiment.

The information processing apparatus 1 according to the first exemplaryembodiment is, for example, assumed to be a mobile information terminal(mobile display device) such as a tablet computer or a smartphone.

The information processing apparatus 1 illustrated in FIG. 1 includessix flat faces, among which a display 2 is arranged on a face. Note thatall of the faces of the information processing apparatus 1 are notnecessarily flat. In other words, one or more faces may be curved.

In the example illustrated in FIG. 1, a face on which the display 2 isprovided is referred to as a front face, faces positioned left and rightof the display 2 are referred to as side faces, and a face opposite tothe display 2 is referred to as a back face or a rear face. In addition,a face positioned above the display 2 is referred to as a top face, anda face positioned below the display 2 is referred to as a bottom face.

In the case of the information processing apparatus 1 illustrated inFIG. 1, the shape of the front face is a rectangle in which a length Hin the Z direction (height) is longer than a length W in the X direction(width). In addition, a length D in the Y direction (depth) defining theside surfaces of the information processing apparatus 1 is shorter thanthe length W in the X direction (width).

The display 2 is configured from a thin film display such as a liquidcrystal display or an organic electroluminescent (EL) display. If thedisplay 2 is a liquid crystal display, a light source (not illustrated)is also provided.

A controller 3 that controls operations of the units including thedisplay 2 and other components (not illustrated) are built in a housingof the information processing apparatus 1.

FIG. 2 illustrates an arrangement example of planar sensors.

FIG. 2 illustrates a position detection sensor 4, a pressure detectionsensor 5, and a distortion detection sensor 6.

The position detection sensor 4 is a sensor that detects the position ofan input operation performed by a user and is stacked on the top face ofthe display 2. An electronic device combining the position detectionsensor 4 with the display 2 is called a touch panel.

The position detection sensor 4 is an example of a detector. Adetectable object differs according to a detection method. For example,in a case of using electrostatic capacitance for detection, parts of aperson's body (e.g., fingertips) are detectable objects. For example, ina case of using infrared rays for detection, fingertips and otherobjects including pens are detectable objects.

The position detection sensor 4 outputs the coordinates of a detectedobject. The position detection sensor 4 according to the first exemplaryembodiment is capable of detecting plural operation positions at a time.

The pressure detection sensor 5 is a sensor that detects the strength ofpressure applied to an operation position at the time of an inputoperation and is provided on the rear face of the display 2, forexample.

The pressure detection sensor 5 according to this exemplary embodimentis a capacitive pressure sensor and detects, as the strength ofpressure, the degree of flexure generated in a body of the sensor formedin the form of a film. In the case of this exemplary embodiment, thepressure detection sensor 5 is capable of detecting some levels ofpressure differences.

The distortion detection sensor 6 is a sensor that detects the degree ofdistortion generated in the body of the sensor and is provided on therear face of the display 2.

As the distortion detection sensor 6 according to this embodiment, forexample, a displacement sensor to which piezoelectricity of polylacticacid is applied, developed by Murata Manufacturing Co., Ltd., is used.In the case of this exemplary embodiment, the distortion detectionsensor 6 is capable of detecting the direction and degree of distortion.The distortion is an example of a physical quantity by which, typically,a part is not directly determined.

The position detection sensor 4, the pressure detection sensor 5, andthe distortion detection sensor 6 are each an example of a detector.

FIG. 3 illustrates an arrangement example of other sensors. Theinformation processing apparatus 1 illustrated in FIG. 3 includes,inside the housing, pressure detection sensors 7A and 7B that detect thestrength of pressure applied locally to lower portions of the sidesurfaces, an acceleration detection sensor 8 that detects the directionand value of acceleration applied to the body, temperature detectionsensors 9A, 9B, and 9C that detect temperatures, and humidity detectionsensors 10A, 10B, and 10C that detect humidities.

The number of temperature detection sensors and the number of humiditydetection sensors may each be one.

The pressure detection sensors 7A and 7B according to this exemplaryembodiment are capable of detecting some strength levels of pressureapplied to the side surfaces of the information processing apparatus 1.

The temperature detection sensors 9A, 9B, and 9C are used for detectionof the temperature (ambient temperature) of the space where theinformation processing apparatus 1 is used and also are used fordetection of the temperature of a local area. The temperature is anexample of a physical quantity by which, typically, a part is notdirectly determined.

The humidity detection sensors 10A, 10B, and 10C are used for detectionof the humidity of the space where the information processing apparatus1 is used and also are used for detection of the humidity of a localarea. The humidity is an example of a physical quantity by which,typically, a part is not directly determined.

The pressure detection sensors 7A and 7B, the acceleration detectionsensor 8, the temperature detection sensors 9A, 9B, and 9C, and thehumidity detection sensors 10A, 10B, and 10C are each an example of adetector.

FIG. 4 illustrates a hardware configuration example of the informationprocessing apparatus 1.

The information processing apparatus 1 according to this exemplaryembodiment includes, in addition to the above-described devices, anon-volatile storage device 14 used for data storage and a communicationunit 15. These devices transmit and receive data via a bus 16.

Note that the controller 3 includes a central processing unit (CPU) 11that executes data processing, a read only memory (ROM) 12 that storesprograms and the like such as basic input/output system (BIOS) andfirmware, and a random access memory (RAM) 13 used as a work area.

The storage device 14 according to this exemplary embodiment isconfigured from, for example, a semiconductor memory or a hard diskdevice.

The communication unit 15 is a communicator used for communication withan external apparatus. A variety of schemes are used for communication.Note that the communication path may be a wired path or a wireless path.

FIG. 5 illustrates a functional configuration example of the controller3 according to the first exemplary embodiment.

The functional configuration illustrated in FIG. 5 is realized by aprogram being executed.

The controller 3 according to the first exemplary embodiment serves as apressure strength determiner 21 that determines the strength ofpressure, a pressurized part determiner 22 that determines the part towhich pressure has been applied, an operation position determiner 23that determines the position of an operation, a temperature determiner24 that determines a temperature, a humidity determiner 25 thatdetermines a humidity, an acceleration direction determiner 26 thatdetermines the direction of acceleration, an acceleration valuedeterminer 27 that determines the value of acceleration, a distortiondeterminer 28 that determines the direction and degree of distortion,and a display content determiner 29 that determines display content byusing determined information.

The display content determiner 29 is an example of a display controller.

The pressure strength determiner 21 receives a pressure value that isoutput from the pressure detection sensor 5 provided on the front faceside and pressure values that are output from the two pressure detectionsensors 7A and 7B provided on the side face side and outputs thestrengths of pressure applied to the respective positions fromcomparison with a threshold that is prepared in advance.

If pressure values are input from the pressure detection sensors 7A and7B provided on the side face side, the pressurized part determiner 22determines that the positions at which pressure is applied by a useroperation are on the side faces. On the other hand, if operationcoordinates are input from the position detection sensor 4, thepressurized part determiner 22 determines that the position at whichpressure is applied by a user operation is on the front face. Althoughthe pressurized part determiner 22 and the operation position determiner23 are provided separately in this exemplary embodiment, the pressurizedpart determiner 22 may also serve as the operation position determiner23.

If pressure values are input from the pressure detection sensors 7A and7B provided on the side face side, the operation position determiner 23determines that a user operates the side surfaces. If operationcoordinates are input from the position detection sensor 4, theoperation position determiner 23 determines that a user operates theposition according to the operation coordinates. The operation positiondeterminer 23 also determines, in addition to operation positions atrespective times, a locus of the operation positions over time. Theoperation position determiner 23 according to this exemplary embodimentis capable of detecting plural operation positions at a time.

The temperature determiner 24 determines temperatures of the respectiveparts, a temperature distribution, a temporal change, and the like onthe basis of temperature values that are input from the temperaturedetection sensors 9A, 9B, and 9C.

The humidity determiner 25 determines humidities of the respectiveparts, a humidity distribution, a temporal change, and the like on thebasis of humidity values that are input from the humidity detectionsensors 10A, 10B, and 10C.

The acceleration direction determiner 26 determines the direction ofacceleration that has acted on the housing and a temporal change thereofon the basis of acceleration information that is input from theacceleration detection sensor 8.

The acceleration value determiner 27 determines the value ofacceleration that has acted on the housing and a temporal change thereofon the basis of acceleration information that is input from theacceleration detection sensor 8.

The distortion determiner 28 determines the direction and degree ofdistortion generated in the housing on the basis of an output from thedistortion detection sensor 6.

On the basis of information from the above-described determiners, thedisplay content determiner 29 makes a change to an object that isdisplayed three-dimensionally on the display 2. Specific change contentswill be specifically described later.

Note that the physical quantity as a detection target differs accordingto the type and arrangement of sensors provided in the housing.

In addition, all of the above-described sensors are not necessarilyprovided in the housing.

Display Control Example

Next, examples of a control operation performed by the display contentdeterminer 29 according to the first exemplary embodiment by usinginputs from sensors will be described.

FIG. 6 illustrates an example of a three-dimensional object displayed onthe display 2 of the information processing apparatus 1.

The three-dimensional object illustrated in FIG. 6 represents acharacter 30 (virtual creature) in the form of a person and is anexample of an object that is displayed three-dimensionally.

FIG. 7 illustrates changes that are made to the character 30 displayedas the three-dimensional object upon acceleration (collision) beingapplied to the top face of the information processing apparatus 1. Theacceleration (collision) is an example of a physical quantity by which,typically, a part is not directly determined.

The example illustrated in FIG. 7 is a case in which acceleration isapplied downward (−Z direction) to the information processing apparatus1 at time t1. This event is determined by the acceleration directiondeterminer 26 (see FIG. 5) if, for example, collision is applied to thetop face of the information processing apparatus 1.

At this time, the display content determiner 29 (see FIG. 5) determinesthe part of the three-dimensional object on which the external forceacts, on the basis of information regarding the determined accelerationdirection.

In the example of FIG. 7, the head of the character 30 is positionedhigher than any other portions. Thus, the display content determiner 29determines that the external force acts on the head of the character 30.

Note that the acceleration value determiner 27 (see FIG. 5) alsodetermines the value of acceleration at time t1. The acceleration valuedeterminer 27 compares a numeric value that is received from theacceleration detection sensor 8 (see FIG. 4) with a threshold, anddetermines the strength of the external force that has acted on theinformation processing apparatus 1.

If the determined value of acceleration is small, the display contentdeterminer 29 changes the display content at time t2 (t2>t1) in such amanner that a small amount of blood 31 flows from the head. The head isan example of a specific part. Note that the number of specific parts isnot necessarily one and may be plural. In addition, the specific partis, in principle, a part of the three-dimensional object.

On the other hand, if the determined value of acceleration is large, thedisplay content determiner 29 changes the display content at time t2(t2>t1) in such a manner that a greater amount of blood 32 flows fromthe head than the amount of a case in which the value of acceleration issmall.

Although the display content in the example of FIG. 7 is changed in sucha manner that the head of the character 30 bleeds upon the detection ofacceleration, a bruise (internal bleeding) may be displayed at the parton which the external force has acted. At this time, the area of thebruise may be changed in accordance with the strength of the externalforce that has acted.

Alternatively, a wen may be displayed at the part on which the externalforce has acted. At this time, the size of the wen may be changed inaccordance with the strength of the external force that has acted.

Although the head of the character 30 is determined as an example of apart at a high position in the direction of acceleration that has actedon the information processing apparatus 1 in the example of FIG. 7, anarm of the character 30 may be determined as a part to which a change isto be made if the acceleration is in the left-right direction of thedisplay 2.

In addition, although the character 30 in the form of a person isassumed as an example of the three-dimensional object in the example ofFIG. 7, the three-dimensional object may be, for example, a structure.If the three-dimensional object is a structure, a change may be made insuch a manner that a scratch or a crack is generated in accordance withthe determined value of acceleration.

FIG. 8 illustrates changes that are made to the character 30 displayedas the three-dimensional object if acceleration (collision) iscontinuously applied to the top face of the information processingapparatus 1.

As a method for continuously applying acceleration (collision), forexample, the information processing apparatus 1 may tap another object,the information processing apparatus 1 may be tapped by another object,or the information processing apparatus 1 may be shaken while beingheld. In the example illustrated in FIG. 8, a case in which a usercontinuously taps an end face (top face) of the information processingapparatus 1 with a finger or the like is assumed.

Thus, the column chart illustrated in FIG. 8 has the horizontal axisrepresenting time and the vertical axis representing the number of timesof collision applied at respective times.

Note that, unlike in the example illustrated in FIG. 8, a part of thedisplayed image may be tapped with a fingertip to apply acceleration(collision) to a specific part. In this case, the position at whichacceleration (collision) is applied is determined by the positiondetection sensor 4 (see FIG. 4). The value of acceleration is detectedby the acceleration detection sensor 8 (see FIG. 4).

In the example of FIG. 8, the number of times of collision increasesfrom time t1 to time t2, and then decreases. Although collisions do notnecessarily have equal strength, typically, the total strength ofcollisions that act on the specific part is in proportion to the numberof times of collision. However, the total strength of five collisionseach having a collision strength of “2” equals to the total strength oftwo collisions each having a collision strength of “5”.

Thus, in the example of FIG. 8, the area of flowing blood 33 increasesfrom time t1 to time t2, and the area of the blood 33 flowing from thehead of the character 30 decreases after time t2 from which the numberof times of collision decreases (time t3) and disappears (time t4).

Note that the total strength is an example of a total of results ofplural times of detection.

FIG. 9 illustrates changes that are made to a structure 34 displayed asthe three-dimensional object if acceleration (collision) is continuouslyapplied to the top face of the information processing apparatus 1.

FIG. 9 illustrates a case in which the structure 34 is a cylinder.

Also in the example illustrated in FIG. 9, acceleration is applieddownward (−Z direction) to the information processing apparatus 1 attime t1. At this time, the display content on the information processingapparatus 1 is changed in such a manner that a scratch 35 is formed onthe top face of the cylinder and a crack 36 grows inside the cylinder.Although the crack 36 generated inside is represented by a dashed linein FIG. 9 for illustration, only the scratch 35 observable from theoutside may be displayed on the information processing apparatus 1.

In FIG. 9, acceleration (collision) is still applied to the top face ofthe information processing apparatus 1 at time t2 (t2>t1). The value ofacceleration applied at time t2 may be equal to or different from thevalue of acceleration applied at time t1.

In the display at time t2, the width of the scratch 35 increases and thecrack 36 grows in the depth direction compared with that in the displayat time t1.

In addition, acceleration (collision) is still applied to the top faceof the information processing apparatus 1 at time t3 (t3>t2). The valueof acceleration applied at time t3 may be equal to or different from thevalue of acceleration applied at time t1 or t2.

In the display at time t3, the scratch 35 is wider and the crack 36 isdeeper than those in the display at time t2.

FIG. 10 illustrates changes that are made to the character 30 displayedas the three-dimensional object upon pressure being applied to the headof the character 30.

In the example illustrated in FIG. 10, a case in which pressure isapplied from the front face side of the information processing apparatus1 toward the rear face side (Y direction) at time t1 is illustrated.

The position on which the pressure acts at this time is determined bythe pressurized part determiner 22 (see FIG. 5) that receives an outputsignal from the position detection sensor 4 (see FIG. 4). This isbecause the position to which the pressure is applied is the same as thecontact position of a fingertip 37.

The strength of pressure at this time is determined by the pressurestrength determiner 21 (see FIG. 5) that receives an output signal fromthe pressure detection sensor 5 (see FIG. 4). The pressure strengthdeterminer 21 compares the numerical value that is input from thepressure detection sensor 5 with a predetermined threshold to determinethe strength of pressure that has acted on the information processingapparatus 1. The strength is determined in some levels.

If the determined strength of pressure is weak, the display contentdeterminer 29 changes the display content at time t2 (t2>t1) in such amanner that a small amount of the blood 31 flows from the head.

On the other hand, if the determined strength of pressure is strong, thedisplay content determiner 29 changes the display content at time t2 insuch a manner that a greater amount of the blood 32 flows from the headthan the amount of a case in which the strength of pressure is weak.

In this example, a user may directly determine the part to which achange is to be made in the display. In addition, in this example, auser may adjust the amount of change to be made to the specific part inaccordance with the strength of pressure.

Although, in the display position of the character 30 displayed on thedisplay 2 in FIG. 10, the position to which pressure is applied isdetermined by using the output from the pressure detection sensor 5, thepart of the character 30 to which a change is to be made in the displaymay be determined by using the outputs from the pressure detectionsensors 7A and 7B (see FIG. 3) that are prepared for detecting locallyapplied pressure.

For example, if pressure is detected by the pressure detection sensors7A and 7B positioned in the lower portions of the left and right sidefaces of the information processing apparatus 1, in accordance with thestrength of the detected pressure, as in FIG. 10, blood may flow fromthe head, or a change may be made to another part such as a hand, a leg,or the chest in the display.

If the output values from the pressure detection sensors 7A and 7B areused, the part to which a change is to be made may be set in the displaycontent determiner 29 in advance.

FIG. 11 illustrates changes that are made to the character 30 displayedas the three-dimensional object if pressure is continuously applied bythe fingertip 37 to the head of the character 30.

Needless to say, the pressure may be applied by an object other than afingertip, such as a tip of a pen. In addition, the fingertip 37 doesnot have to be continuously in contact with the surface of the display 2and may tap a specific part discontinuously as in the case in whichacceleration (collision) is applied.

In the example illustrated in FIG. 11, the head of the character 30remains pushed by the fingertip 37 from time t1 to time t3. In thiscase, the display content determiner 29 determines that the head keepsbleeding and adds an image of the blood 31 to the head at time t2 andincreases the area of the blood 32 at time t3.

Here, the bleeding amount may be in proportion to the strength ofpressure. For example, if the strength is strong, it may be determinedthat the bleeding amount is large; if the strength is weak, it may bedetermined that the bleeding amount is small. However, regardless of thestrength of pressure, the bleeding amount may be increased and decreasedin proportion to the length of time during which a pressure exceeding apredetermined threshold is applied.

Although the above-described example is a case in which a change made ona screen is bleeding or a scratch (crack) depending on a difference ofthe three-dimensional object displayed on the display 2, the changecontent may be different in accordance with the part on which anexternal force acts. Note that the change content may be any content aslong as the specific part is displayed in a different form, such asaddition, deletion, deformation, or division in the display.

FIG. 12 illustrates changes that are made to the character 30 displayedas the three-dimensional object upon pressure being applied to the chestof the character 30.

In the example illustrated in FIG. 12, the chest, not the head, ispushed by the fingertip 37 at time t1. In this example, a change is madeto clothing depending on a difference of the strength of pressureapplied to the chest.

If the determined strength of pressure is weak, the display contentdeterminer 29 changes the image into an image in which the clothing ofthe character 30 is lifted or strained at time t2 (t2>t1).

On the other hand, if the determined strength of pressure is strong, thedisplay content determiner 29 changes the image into an image in whichthe clothing of the character 30 is ripped at time t2.

In addition, for example, the type of clothing may be changed or theclothing may be changed depending on the determined strength ofpressure.

FIG. 13 illustrates changes that are made to the structure 34 displayedas the three-dimensional object upon distortion being applied to theinformation processing apparatus 1.

FIG. 13 illustrates a cylinder as the structure 34. Stripes are providedon the outer circumferential face of the cylinder. The stripes are inparallel to the axial direction of the rotationally symmetric cylinder.

In the example illustrated in FIG. 13, while a counterclockwise forceacts on an upper portion of the information processing apparatus 1 inthe form of a flat plate, a clockwise force acts on a lower portion ofthe information processing apparatus 1. That is, in FIG. 13, oppositeforces act on the upper portion and the lower portion of the informationprocessing apparatus 1.

At this time, the direction and degree of distortion acting on theinformation processing apparatus 1 are given from the distortiondeterminer 28 (see FIG. 5) to the display content determiner 29.

In accordance with the direction and degree of distortion determined attime t1, the display content determiner 29 generates an image in whichthe structure 34 is distorted and displays the image on the display 2.

In the example of FIG. 13, a change is made to the structure 34 at timet2 at which distortion is no longer applied, not at time t1 at whichdistortion is applied to the information processing apparatus 1.However, a change may be made to the displayed structure 34 at time t1at which distortion is applied.

Although the change content works together with the distortion appliedto the information processing apparatus 1 in the example of FIG. 13,another change may be made to the three-dimensional object, which is adisplay target, by using information on the determined direction anddegree of distortion. For example, a character displayed on the display2 may bleed or be scratched, or a change may be made to the clothing ofthe character.

FIG. 14 illustrates changes that are made to the character 30 displayedas the three-dimensional object upon a temperature change being detectedby the temperature detection sensors 9A, 9B, and 9C (see FIG. 3).

In this case, the display content determiner 29 makes a change to aspecific part of the displayed character 30 on the basis of a result ofcomparison between a determined temperature and a predeterminedthreshold or information on the distribution of temperatures detected bythe plural temperature detection sensors 9A, 9B, and 9C.

If, for example, the ambient temperature of the information processingapparatus 1 detected at time t1 is lower than a predetermined threshold,the display content determiner 29 changes the display content at time t2into a state in which a scarf 38 is put around the neck of the character30. The neck of the character 30 is an example of the specific part.

If, for example, the ambient temperature of the information processingapparatus 1 detected at time t1 is higher than a predeterminedthreshold, the display content determiner 29 changes the display contentat time t2 into a state in which the forehead of the character 30 iscovered with sweat 39. The forehead of the character 30 is an example ofthe specific part.

FIG. 15 illustrates a change that is made to an ice cube 40 displayed asthe three-dimensional object upon a temperature change being detected bythe temperature detection sensors 9A, 9B, and 9C (FIG. 3).

In this case, the display content determiner 29 makes a change to aspecific part of the displayed ice cube 40 on the basis of a result ofcomparison between a determined temperature and a predeterminedthreshold or information on the distribution of temperatures detected bythe plural temperature detection sensors 9A, 9B, and 9C.

The example of FIG. 15 is a case in which the ambient temperature of theinformation processing apparatus 1 is higher than the predeterminedthreshold. In this case, the display content determiner 29 makes achange to in such a manner that the displayed ice cube 40 melts to besmaller and water 41 is gathered around the ice cube 40. The ice cube 40is an example of the specific part.

FIG. 16 illustrates a change that is made to the character 30 displayedas the three-dimensional object upon a humidity change being detected bythe humidity detection sensors 10A, 10B, and 10C (see FIG. 3).

In this case, the display content determiner 29 makes a change to aspecific part of the displayed character 30 on the basis of a result ofcomparison between a determined humidity and a predetermined thresholdor information on the distribution of humidities detected by the pluralhumidity detection sensors 10A, 10B, and 10C.

The example of FIG. 16 is a case in which the ambient humidity of theinformation processing apparatus 1 is higher than the predeterminedthreshold. In this case, the display content determiner 29 changes theclothing and footwear of the character 30 into a raincoat 43 and rainboots 42.

The body and legs of the character 30 are each an example of thespecific part. In addition, the clothing and footwear may be changed tothe raincoat 43 and the rain boots 42 for rainy weather. Note that theclothing and footwear may be changed to those for waterside such asriver or sea.

Other Configurations

Although the single acceleration detection sensor 8 detects the value ofacceleration applied to the display 2 (see FIG. 1) in the firstexemplary embodiment, plural acceleration detection sensors 8 may beprovided in the plane of the display 2 as illustrated in FIG. 17.

FIG. 17 illustrates an example in which the acceleration detectionsensors 8 are arranged in M rows and N columns in the plane of thedisplay 2. In the case in which the acceleration detection sensors 8 arearranged as illustrated in FIG. 17, the distribution of values of localacceleration may be determined from positional information of theacceleration detection sensors 8.

FIG. 18 illustrates a change in the number of times of detection ofacceleration (collision) from a specific position in a case in which theacceleration detection sensors 8 are arranged in 4 rows and 4 columns.

In the case of FIG. 18, no collision is applied at time t1. At time t1,the number of times of detection is zero in each of the accelerationdetection sensors 8. Collision is applied once to two accelerationdetection sensors 8 positioned in the first row and the second and thirdcolumns at time t2. Collision is applied 10 times to the twoacceleration detection sensors 8 positioned in the first row and thesecond and third columns at time t3, and propagation of vibrations inthis process changes outputs from the acceleration detection sensors 8in the second and third rows in the same columns. In the illustration ofFIG. 18, darker shading of the acceleration detection sensors 8indicates a larger total number of times of collision.

In the example at time t3, it is found that acceleration propagates fromthe first row to the second and third rows on the basis of thedistribution of the number of times of collision detected by the pluralacceleration detection sensors 8 within a predetermined period.Information on this propagation direction may be used to control contentof a change to be made to the specific part.

At time t4, five minutes elapses after collision has no longer beenapplied. In this state, each number of times of detection of collisionis reset to zero.

Note that substantially the same technique may be applied to thepressure detection sensor 5. The position detection sensor 4 and thepressure detection sensor 5 having substantially the same shape as thedisplay 2 (see FIG. 2) are combined to detect the position and strengthof pressure in the first exemplary embodiment. However, plural pressuredetection sensors 5 may be arranged in the plane of the display 2, andpositional information of the pressure detection sensors 5 that detectpressure may be used to calculate the position at which pressure isapplied, the direction in which pressure as an external force isapplied, the strength thereof, and the like.

In addition, a part to which a change is to be made in the display maybe determined on the basis of the change and distribution of strengthsof pressure detected by the respective pressure detection sensors 5.

In addition, the first exemplary embodiment has illustrated a case inwhich the three-dimensional object is displayed on the single display 2(see FIG. 1) included in the information processing apparatus 1 (seeFIG. 1). However, as illustrated in FIG. 19, substantially the sametechnique may be applied to processing of the controller 3 in aninformation processing apparatus 1A that displays an objectthree-dimensionally by displaying plural two-dimensional objectscorresponding to a single object.

FIG. 19 illustrates an appearance example of the information processingapparatus 1A including displays 2A to 2D on four faces, which are thefront face, the side faces, and the back face.

In FIG. 19, a character front image 30A is displayed on the display 2Aprovided on the front face, a character right side image 30B isdisplayed on the display 2B provided on the right side face of thedrawing, a character left side image is displayed on the display 2Cprovided on the left side face, and a character back image is displayedon the display 2D provided on the back face.

Note that the displays may be provided on two faces, such as the frontand back faces or the top and bottom faces.

Also in a case of displaying an object three-dimensionally by arrangingplural displays 2 three-dimensionally as in the above case, by making achange to a specific part of the displayed object on the basis ofoutputs from the sensors, it is possible to provide new representationto a user.

Second Exemplary Embodiment Apparatus Configuration

FIG. 20 illustrates an appearance example of an information processingapparatus 1B according to a second exemplary embodiment.

In FIG. 20, parts corresponding to those in FIG. 1 are denoted by thecorresponding reference numerals.

The appearance of the information processing apparatus 1B issubstantially in the form of a plate as in the appearance of theinformation processing apparatus 1 according to the first exemplaryembodiment.

However, the information processing apparatus 1B according to the secondexemplary embodiment is different from the information processingapparatus 1 according to the first exemplary embodiment in that ahousing is deformable at any position and in including deformationdetection sensors 51 that detect the position where the housing isdeformed.

The plural deformation detection sensors 51 are arranged along theoutline of the housing. The arrangement positions and intervals(density) of the deformation detection sensors 51 are determined inaccordance with the size or specifications of the deformation detectionsensors 51. Note that the deformation detection sensors 51 may bearranged to overlap with the display 2.

Each of the deformation detection sensors 51 is configured from aso-called strain sensor, and, for example, a displacement sensor towhich piezoelectricity of polylactic acid is applied, developed byMurata Manufacturing Co., Ltd., is used. The strain sensor outputs asensor output whose level is in accordance with a bent amount (angle).Thus, the strain sensor is a device by which deformation of an attachedmember is detectable. Accordingly, the deformation detection sensors 51also detect curving before bending as deformation.

Note that a state in which the housing of the information processingapparatus 1B is planar is referred to as a pre-deformation state or aninitial state in the second exemplary embodiment.

On the basis of a positional relationship between the plural deformationdetection sensors 51 that detect bending, the controller 3 estimates abent position.

In this exemplary embodiment, the deformation detection sensors 51 arearranged on one face (e.g., a face on which the display is provided) ofthe housing. However, the deformation detection sensors 51 may bearranged on both faces of the housing.

The bent position is an example of a deformed position.

In the information processing apparatus 1B, a flexible housing formed ofplastic or the like is provided with the display 2 used to display animage, the controller 3 that controls the entire apparatus, and thelike. The information processing apparatus 1B that is specialized indisplaying is referred to as a flexible display.

FIG. 21 illustrates a hardware configuration example of the informationprocessing apparatus 1B.

In FIG. 21, parts corresponding to those in FIG. 4 are denoted by thecorresponding reference numerals. The information processing apparatus1B is different from the information processing apparatus 1 according tothe first exemplary embodiment in that the plural deformation detectionsensors 51 are connected to the bus 16.

FIG. 22 illustrates a functional configuration example of the controller3 according to the second exemplary embodiment.

In FIG. 22, parts corresponding to those in FIG. 5 are denoted by thecorresponding reference numerals. The information processing apparatus1B is different from the information processing apparatus 1 described inthe first exemplary embodiment in including a bent position determiner52 that receives outputs from the plural deformation detection sensors51 and determines a bent position generated in the housing.

The display content determiner 29 according to this exemplary embodimentuses information on a bent position in the housing determined by thebent position determiner 52 to make a change to a specific part of anobject that is displayed three-dimensionally or to make a change todisplay content.

FIGS. 23A to 23E illustrate examples of the bent position determined bythe bent position determiner 52.

FIG. 23A illustrates a bent position determined by the bent positiondeterminer 52 if deformation is detected by two deformation detectionsensors 51 each positioned near the midpoint of a short side of theinformation processing apparatus 1B. In the case of FIG. 23A, the bentposition determiner 52 determines that the display 2 is bent in such amanner that a crease is put along a line L1 in parallel to long sides ofthe information processing apparatus 1B. Note that creases in FIGS. 23Ato 23E are illustrated for description, and the creases are notnecessarily put.

FIG. 23B illustrates a bent position determined by the bent positiondeterminer 52 if deformation is detected by two deformation detectionsensors 51 each positioned near the midpoint of a long side of theinformation processing apparatus 1B. In the case of FIG. 23B, the bentposition determiner 52 determines that the display 2 is bent in such amanner that a crease is put along a line L2 in parallel to short sidesof the information processing apparatus 1B.

FIG. 23C illustrates a bent position determined by the bent positiondeterminer 52 if deformation is detected by two deformation detectionsensors 51 positioned at the upper right corner and the lower leftcorner of the information processing apparatus 1B. In the case of FIG.23C, the bent position determiner 52 determines that the display 2 isbent in such a manner that a crease is put along a line L3 in a diagonalline to the upper right corner.

FIG. 23D illustrates a bent position determined by the bent positiondeterminer 52 if deformation is detected by a deformation detectionsensor 51 positioned at the upper right corner and a deformationdetection sensor 51 positioned immediately below a deformation detectionsensor 51 positioned at the upper left corner of the informationprocessing apparatus 1B. In the case of FIG. 23D, the bent positiondeterminer 52 determines that the display 2 is bent in such a mannerthat a crease is put along a line L4 that forms the hypotenuse of aright triangle whose right angle is at the upper left corner.

FIG. 23E illustrates a bent position determined by the bent positiondeterminer 52 if deformation is detected by three deformation detectionsensors 51 positioned in the upper side (long side) and threedeformation detection sensors 51 positioned in the lower side (longside). In the case of FIG. 23E, the bent position determiner 52determines that the display 2 is deformed in such a manner that creasesare put along a line L5 in a valley fold, a line L6 in a mountain fold,and a line L7 in a valley fold from the left side to the right side.

Display Control Example

Next, examples of a control operation performed by the display contentdeterminer 29 according to the second exemplary embodiment by usinginputs from sensors will be described.

FIG. 24 illustrates an example in which an image of thethree-dimensional object is edited by using information on thedetermined bent position.

In FIG. 24, at time t1 before the information processing apparatus 1B isdeformed, a three-dimensional image 53 of the three-dimensional objectis displayed on the display 2. The three-dimensional image 53 in thiscase is an apple (virtual creature). In addition, the shape of theinformation processing apparatus 1B at time t1 is an example of a firstshape.

At the next time, time t2, the information processing apparatus 1B isbent at a relatively right position with respect to the center of thescreen in such a manner that the display 2 comes inside. Also at thistime, no change is made to the displayed three-dimensional image 53 ofthe three-dimensional object. The shape of the information processingapparatus 1B at time t2 is an example of a second shape.

At the next time, time t3, the information processing apparatus 1B ismade flat again, but a line 54 is added to the three-dimensional image53 of the three-dimensional object at a part corresponding to the bentposition at time t2 (curved part on the surface of the apple). In otherwords, the line addition is an example of editing on thethree-dimensional image 53 (e.g., image itself) of the three-dimensionalobject displayed across the bent position.

In the case of FIG. 24, a user operation for bending the informationprocessing apparatus 1B is used for the display content determiner 29 toadd the line 54. Although the line 54 is started to be displayed afterthe information processing apparatus 1B has been made flat again (at andafter time t3) in the example of FIG. 24, the line 54 may be started tobe displayed while the information processing apparatus 1B is bent (attime t2). In this case, the displaying of the line 54 may end when theinformation processing apparatus 1B is no longer bent.

Here, the part at which a dashed line corresponding to the bent positioncrosses the three-dimensional image 53 of the three-dimensional objectcorresponds to a specific part.

FIG. 25 illustrates another example in which an image of thethree-dimensional object is edited by using information on thedetermined bent position.

Also in FIG. 25, no change is made to the three-dimensional image 53 ofthe three-dimensional object displayed on the display 2 regardless ofthe bending state of the information processing apparatus 1B at time t1and time t2.

However, in the case of FIG. 25, after the information processingapparatus 1B has been made flat again, an image in which thethree-dimensional image 53 of the three-dimensional object is cut at apart corresponding to the bent position at time t2 is displayed as thethree-dimensional image 53 of the three-dimensional object. That is, acut plane 55 is added.

Although only an image obtained after cutting the three-dimensionalimage 53 of the three-dimensional object is displayed in the example ofFIG. 25, a process from the start to the end of cutting may be displayedas a moving image.

The displaying of the cut plane increases the presence of imageprocessing. In the case of this exemplary embodiment, special effectprocessing to be performed is specified by a user in advance.

Note that sound effects may be added in accordance with the content ofprocessing at the time of image processing. For example, in the exampleof FIG. 25, the sound of cutting an apple may be produced as a soundeffect from a speaker (not illustrated). The addition of sound effectsalso increases the presence of image processing.

Other Configurations

Deformation of the information processing apparatus 1B is also usable tocontrol a processing operation of another information processingapparatus.

FIG. 26 illustrates an example in which a deformation operation of theinformation processing apparatus 1B is used to control a displayoperation of a display device 57 whose display is controlled by anotherinformation processing apparatus, an information processing apparatus56.

In the case of FIG. 26, deformation information detected by theinformation processing apparatus 1B that is deformable at any positionis transmitted to the information processing apparatus 56 through acommunication unit, and is used to control a screen displayed on thedisplay device 57 provided on or connected to the information processingapparatus 56.

The information processing apparatus 56 is configured as a so-calledcomputer, and content of an image displayed on the informationprocessing apparatus 1B used as an operation unit may be the same as ordifferent from content of an image displayed on the display device 57through the information processing apparatus 56.

In the case of FIG. 26, on the basis of deformation information detectedby the information processing apparatus 1B from time t1 to time t2, thecut plane 55 is displayed in a part of the three-dimensional image 53 ofthe three-dimensional object displayed on the display device 57. Inother words, a part of the three-dimensional image 53 is deleted.

FIG. 27 illustrates another example in which a deformation operation ofthe information processing apparatus 1B is used to control a displayoperation of the display device 57 whose display is controlled byanother information processing apparatus, which is the informationprocessing apparatus 56.

FIG. 27 illustrates an example in which the information processingapparatus 56 detects deformation of the information processing apparatus1B as an operation input unit and is used to issue an instruction forstarting a slide show or for turning pages displayed on the displaydevice 57.

In the case of FIG. 27, on the basis of deformation information detectedby the information processing apparatus 1B from time t1 to time t2, ascreen for a slide show is switched from a first page to a second page.In other words, a display image is replaced.

Third Exemplary Embodiment

This exemplary embodiment will describe a case in which athree-dimensional display is used to display an objectthree-dimensionally.

FIG. 28 illustrates an example of a three-dimensional object displayedby a three-dimensional display.

An information processing system 61 illustrated in FIG. 28 includes animage capturing apparatus 64 that captures an image of a user, aninformation processing apparatus 65, and a three-dimensional spacedepicting apparatus 66.

The image capturing apparatus 64 is an apparatus that captures an imageof the movement of a user 63 as a subject and is a type of sensor.

The information processing apparatus 65 is an apparatus that performsprocessing for outputting data of the three-dimensional object that is adisplay target to the three-dimensional space depicting apparatus 66,processing for determining the movement of the user 63 by processing thecaptured image, image processing on the three-dimensional object inaccordance with the determination results, and the like. The informationprocessing apparatus 65 is configured as a so-called computer, and theabove-described various kinds of processing are performed by executionof a program.

The three-dimensional space depicting apparatus 66 is configured from,for example, an infrared pulse laser, a lens for adjusting a focal pointat which an image is formed with laser beams, a Galvanometer mirror usedfor planar scanning of laser beams in a space, and the like and depictsa three-dimensional image in the air on the basis of the given data ofthe three-dimensional object. In the example of FIG. 28, athree-dimensional image 62 of an apple is formed by air plasma emissionand is floating in the air as the three-dimensional object.

Although the movement of the user 63 is detected through imageprocessing in this exemplary embodiment, the movement of the user 63 maybe detected by using an output from a sensor that detects the movementof the air. Not only the movement of the user 63, but also the bodytemperature of the user 63, a change thereof, the ambient temperature ofthe user 63, humidity around the user 63, and a change thereof may bedetected through thermography.

FIG. 29 illustrates a state in which a change is made to a displayedthree-dimensional object in response to a specific movement of the user.

In FIG. 29, parts corresponding to those in FIG. 28 are denoted by thecorresponding reference numerals.

FIG. 29 illustrates a case in which a user moves to vertically cut aspecific part of the three-dimensional image 62. The informationprocessing apparatus 65 that detects the movement of the user determinesthe position of the three-dimensional image 62 to which a change is tobe made in response to the movement of the user, and determines a changecontent. In the case of FIG. 29, the volumes of divisions obtained bydividing the three-dimensional image 62 of the apple along thedetermined position are calculated from data of the three-dimensionalobject, and depiction data is generated such that a division having thelarger volume is depicted in the air.

Thus, in FIG. 29, a cut plane 67 of a division having the larger volumein the three-dimensional image 62 of the apple is depicted.

Although the cut plane 67 is depicted in this example, the changecontent may be controlled in any manner in accordance with thethree-dimensional object that is depicted as in the case of the firstexemplary embodiment.

FIG. 30 illustrates an example in which a change is made to athree-dimensional object that is projected onto a wall or a floor.

In FIG. 30, parts corresponding to those in FIG. 28 are denoted by thecorresponding reference numerals.

An information processing system 61A illustrated in FIG. 30 is differentfrom the information processing system 61 in that a projecting apparatus68 is used in place of the three-dimensional space depicting apparatus66.

In the case of FIG. 30, the projecting apparatus 68 projects athree-dimensional image 69 of the three-dimensional object onto a wall,which is a two-dimensional space. A user illustrated in FIG. 30 moveshis/her right hand from up to down (−Z direction) along the wall, whichis a projection plane. The information processing apparatus 65 detectsthe movement of the user through an infrared sensor (not illustrated) orimage processing and makes a change to the three-dimensional image 69 tobe projected. In the case of FIG. 30, a cut plane 70 is displayed.

Other Exemplary Embodiments

The exemplary embodiments of the present invention have been describedabove. However, the technical scope of the present invention is notlimited to the above-described exemplary embodiments. It is apparentfrom the claims that various modifications or improvements of theabove-described exemplary embodiments are also included in the technicalscope of the present invention.

For example, the color of a specific part of an object that is displayedthree-dimensionally may be changed.

In the above-described exemplary embodiments, to a specific part of anobject that is displayed three-dimensionally on a display, a change ismade by using a sensor such as a sensor that detects the position on adisplay screen receiving user operations (the position detection sensor4), a sensor that detects a user operation as pressure (the pressuredetection sensor 5, 7A, 7B), or a sensor that detects distortion of ahousing (the distortion detection sensor 6 (see FIG. 4)). However, anysensor other than the above-described sensors may be used.

For example, a sensor that measures the elevation of the position wherethe information processing apparatus 1 is used may be used. Examples ofthis type of sensor include an altimeter that measures atmosphericpressure to calculate the elevation, a global positioning system (GPS)receiver that calculates the elevation by using GPS signals, and thelike. Note that some GPS signals are supposed to be used indoors.

Examples further include an air pressure sensor, an illuminance sensor,a water pressure sensor, a water depth sensor, and the like formeasuring the position where the information processing apparatus 1 isused.

In a case of using a sensor that measures the elevation, in accordancewith the elevation of the position where the information processingapparatus 1 is used, a change may be made to a specific part or abackground of an object displayed on the display 2, or a screendisplayed on the display 2 may be changed. For example, at a positionwhere the elevation is high, wings may be added to a part of the object(e.g., the back or arms of a character), clouds may be added to thebackground of the object, or the screen may be switched to a bird's-eyeview looking down upon surrounding landscapes from above or an image ofa sky.

In a case of using a sensor that measures the air pressure, a change maybe made in such a manner that a part of the object (e.g., the stomach orcheeks of a character) is puffed out or sucked in in accordance with achange in the air pressure value. Alternatively, a change may be made toan image in such a manner that the volume of an object representing asealed bag, a balloon, or the like is increased or decreased as the airpressure is decreased or increased.

In a case of using an illuminance sensor, a change may be made to a partof an object in accordance with a change in the illuminance value. Forexample, a character as the object may wear sunglasses in brightlocations or may carry a flashlight in dark locations. In addition, thedisplay form of the object or the display content of the screen may beswitched between night and day in accordance with the brightness of anambient environment.

Furthermore, a GPS sensor may be used as the sensor. For example, in acase in which an operation manual of an apparatus is displayed as theobject, the language therein may be changed to the first language of thedetected country or area.

Contents of questions and answers described in the operation manual orcontents related to precautions for operation may be changed on thebasis of an output value of a sensor, which changes in accordance with ause environment of the information processing apparatus 1. The changeherein includes, for example, a change of the position of description insuch a manner that contents that are likely to be referred to in anenvironment determined on the basis of an output from a sensor areplaced in higher levels.

For example, if it is determined that the information processingapparatus 1 is used in a hot and humid environment, a change may be madein such a manner that precautions for a hot and humid environment areplaced in higher levels of the operation manual displayed as the object.

Alternatively, for example, pictures in the operation manual may belocalized (customized) in accordance with a hot and humid environment.

The above exemplary embodiments have described a case in which eachsensor is provided in the information processing apparatus 1. However, asensor may be independent of the information processing apparatus 1, andan output value of the sensor may be given to the information processingapparatus 1 through a communication unit.

In addition, a change may be made to an object displayed on theinformation processing apparatus 1 on the basis of environmentalinformation (first information) acquired from the outside. Theenvironmental information herein is information that is related topositional information of the information processing apparatus 1 andthat is acquirable from the outside and includes, for example,information regarding weather, such as a weather forecast, informationregarding crime prevention, such as occurrence of a crime, andinformation regarding traffic, such as a traffic accident or a trafficjam.

For example, in a state in which a map is displayed on a display screenof the information processing apparatus 1 and in which a character imagerepresenting the position of a user (i.e., the position of theinformation processing apparatus 1) is displayed on the map, uponreception of a high-temperature warning associated with the position ofthe user, sleeves and cuffs of the character image may be shortened(clothing may be changed to a short-sleeved shirt and shorts), and thebody may be sweating.

For example, in a state in which a map is displayed on a display screenof the information processing apparatus 1 and in which a character imagerepresenting the position of a user (i.e., the position of theinformation processing apparatus 1) is displayed on the map, uponreception of a notification of occurrence of a crime in an areaassociated with the position of the user, a change may be made in such amanner that the character's expression is changed to a frightenedexpression or the character's body is trembling.

For example, upon reception of a notification of occurrence of a trafficaccident, a change may be made to the display in the same manner.

Note that if plural pieces of environmental information are acquiredtogether, the display content may be changed by combining changescorresponding to the plural pieces of environmental information. Forexample, if the above high-temperature warning and the abovenotification of occurrence of a crime are both acquired, a change may bemade in such a manner that a character wears light clothing, issweating, and has a frightened expression.

In addition, a change may be made to the displayed object by combininginformation acquired by the sensor and environmental information.

In the above-described exemplary embodiments, in principle, a specificchange is made to the display by using a physical quantity measured byeach sensor. However, outputs from plural sensors may be combined tomake a change to the display. The physical quantity herein may includepressure, acceleration, temperature, humidity, air pressure, elevation,water depth, magnetic pole, sound, positional information, and the like,which are measurement targets. The physical quantity herein may furtherinclude a change in an electrical signal (current or voltage) thatappears in the sensor.

For example, four pieces of information, which are temporal information,an elevation value, a temperature value, and an illuminance, may becombined to determine a use environment of the information processingapparatus 1, and in accordance with the determined use environment(e.g., mountain in summer), a change may be made to a part of the objector to a screen.

Alternatively, for example, the clothing of a character as the objectmay be changed to clothing in a summer resort or clothing for climbing amountain in summer, and an image displayed on the display 2 may bechanged to an image of a summer sky.

Further alternatively, creatures that live in water areas correspondingto the determined water depth and water temperature may be displayed onthe display 2 by combining the water depth and the temperature value,and a change may be made to the display form of the object.

In the above-described exemplary embodiment, a character, an apple, andthe like are used as examples of the three-dimensional object. However,the object that is displayed three-dimensionally is not limited tothese. For example, an hourglass representing the lapse of time and aset of antennas representing the intensity of radio waves are alsoexamples of the three-dimensional object.

In addition, the three-dimensional object as a display target may be aproduct such as a built-in device, equipment, or a machine that operatesin accordance with software such as firmware. In this case, if designinformation of the product (e.g., computer-aided design (CAD) data orperformance information of constituents) is available, such informationand output values from sensors may be combined in the display.

For example, if the thermal expansion coefficient or the like of eachcomponent to be configured or each member to be attached or detached isacquirable by the information processing apparatus 1 and temperatureinformation is acquired from a sensor, the shape of each component of adisplayed object may be changed in accordance with the acquiredinformation. In this case, the changed shape may be emphasized in thedisplay.

Such a display function enables a change in the display in such a mannerthat the shape of some of the components is expanded if the ambienttemperature of the information processing apparatus 1 is high. Thisdisplay equals to a simulation of a future change of a product under thecurrent environment. Although typical simulations require inputs oftemperature conditions and the like, this display function enableschecking of a change to be generated in the product by only displayingthe target product on a screen on-site.

In addition, if the ambient temperature of the information processingapparatus 1 is low, this display function enables estimation ofincreased viscosity of a lubricant and a change of the movement of aproduct displayed on a screen as a change to hardware.

With this display function, an abnormality of hardware movement due toan influence of a use environment on software or a malfunction ofsoftware may be represented as a change to a displayed object. Forexample, if the ambient environment of the information processingapparatus 1 is humid and hot, it is possible to estimate a short circuitof an electrical circuit configuring an object or thermal runaway of aCPU and to change the movement of the object on a display screen tomovement that is unique to malfunction (e.g., screen blackout or errormessage display).

Note that the change of a displayed object based on information acquiredby various sensors may be realized by using, for example, acorrespondence table 60 illustrated in FIG. 31.

FIG. 31 illustrates an example of the correspondence table 60representing a correspondence relationship between objects as changetargets in combination with numbers of dimensions of sensor values andchanged images.

The correspondence table 60 includes objects as change targets displayedas, in addition to three-dimensional images, one-dimensional images andtwo-dimensional images. That is, although the above exemplaryembodiments have described cases in which a change is made to displayedthree-dimensional objects, the number of dimensions of an object as adisplay target may be any number as long as the display is capable ofdisplaying the object. In addition, if changed images corresponding tothe respective numbers of dimensions are prepared, displayed images maybe changed on the basis of outputs from various sensors or the like.

On the other hand, sensor values are classified according to the numberof dimensions of acquired information. Specifically, one-dimensionalsensor values 72 correspond to cases in which sensor values areone-dimensional, two-dimensional sensor values 73 correspond to cases inwhich sensor values are two-dimensional, and three-dimensional sensorvalues 74 correspond to cases in which sensor values arethree-dimensional.

The classification into the one-dimensional sensor values 72, thetwo-dimensional sensor values 73, and the three-dimensional sensorvalues 74 herein may be based on the number of dimensions in athree-dimensional space, for example.

For example, if a physical quantity detected by a single sensorcorresponds to one dimension (e.g., X-axis direction) in athree-dimensional space, an output value of this sensor is included inthe one-dimensional sensor values 72. Note that, even if two sensorsdetect different physical quantities, as long as the respective physicalquantities correspond to the same dimension (e.g., X-axis direction),output values of the two sensors are the one-dimensional sensor values72.

Similarly, if, for example, a physical quantity detected by a singlesensor corresponds to two dimensions (e.g., X-axis direction and Y-axisdirection) in a three-dimensional space, an output value of this sensoris included in the two-dimensional sensor values 73. In addition, if twosensors detect one-dimensional physical quantities in differentdirections (e.g., one is X-axis direction and the other is Y-axisdirection), output values of the two sensors are the two-dimensionalsensor values 73.

Similarly, if, for example, a physical quantity detected by a singlesensor corresponds to three dimensions (e.g., X-axis direction, Y-axisdirection, and Z-axis direction) in a three-dimensional space, an outputvalue of this sensor is included in the three-dimensional sensor values74. In addition, if three sensors detect one-dimensional physicalquantities in different directions (e.g., one is X-axis direction,another one is Y-axis direction, and the other is Z-axis direction),output values of the three sensors are the three-dimensional sensorvalues 74.

Alternatively, the classification into the one-dimensional sensor values72, the two-dimensional sensor values 73, and the three-dimensionalsensor values 74 herein may be based on the number of dimensions ofinformation, for example.

For example, if a change is to be made to a displayed object inaccordance with a combination of output values of two sensors thatoutput a single value, the two sensor values are the two-dimensionalsensor values 73. In addition, if a single sensor outputs two values,the two sensor values output from the single sensor are thetwo-dimensional sensor values 73. The plural output values herein maycorrespond to the same physical quantity or different physicalquantities.

Further alternatively, the classification into the one-dimensionalsensor values 72, the two-dimensional sensor values 73, and thethree-dimensional sensor values 74 may be based on a combination of thenumber of dimensions of a three-dimensional space and the number ofdimensions of information, for example.

In the example of FIG. 31, plural changed images 1 to N with respect toimages of the respective numbers of dimensions are associated with thelevels of sensor values of the respective numbers of dimensions.

For example, for the one-dimensional sensor values 72, in accordancewith differences of the levels, N changed images 1 are associated withone-dimensional images, N changed images 4 are associated withtwo-dimensional images, and N changed images 7 are associated withthree-dimensional images.

For example, for the two-dimensional sensor values 73, in accordancewith differences of combinations of the levels, M changed images 2 areassociated with one-dimensional images, M changed images 5 areassociated with two-dimensional images, and M changed images 8 areassociated with three-dimensional images.

Similarly, for the three-dimensional sensor values 74, in accordancewith differences of combinations of the levels, L changed images 3 areassociated with one-dimensional images, L changed images 6 areassociated with two-dimensional images, and L changed images 9 areassociated with three-dimensional images.

Although the same number of changed images are associated for eachnumber of dimensions of sensor values regardless of the number ofdimensions of images in the example of FIG. 31, different numbers ofchanged images may be assigned in accordance with combinations. Inaddition, the number of changed images may be determined in accordancewith, instead of the number of dimensions of sensor values, the numberof dimensions of images.

The above exemplary embodiments have described cases in which a singleobject is displayed on a screen for simplicity of description. However,plural objects may be displayed. In this case, it is possible to depictplural objects in the same dimension or in different dimensions. Forexample, if it is possible to depict images in three dimensions, someobjects may be displayed three-dimensionally, and the other objects maybe displayed two-dimensionally or one-dimensionally. Note that all ofthe objects may be displayed in dimensions whose number is smaller thanthe number of possible dimensions.

In this case, a change may be made to each image in accordance with thenumber of dimensions used for depicting the image.

In addition, although a change in accordance with a sensor value may bemade to each image corresponding to one of plural objects, the changemay be made only to a specific object specified by a user among theplural objects or an image corresponding to a specific dimension ordimensions. For example, if there are both three-dimensionally depictedobject images and two-dimensionally depicted object images, a change maybe made only to three-dimensionally depicted object images, which arespecified by a user.

The above exemplary embodiments have described cases in which changesare made to the display of the information processing apparatus 1.However, an output to an image forming apparatus (so-called printer) mayreflect such a change in the display. For example, if a printinstruction is issued in a state in which a change is made to adisplayed three-dimensional object in accordance with a sensor outputvalue, print data corresponding to the changed content may be generatedto print an image. That is, not only image display, but also printingmay reflect a sensor output value.

Note that a change in the display may be independent of content of theimage to be printed. That is, even if a change is made to a displayedobject, content to be printed may be content of the three-dimensionalobject before a change is made in accordance with a sensor output value.In this case, it is desirable that a user be allowed to select on a userinterface screen, whether an image including display content that hasbeen changed in accordance with the sensor output value is to be printedor an image including content before a change is made in accordance withthe sensor output value is to be printed.

The user interface screen herein may be prepared as a setting screen ora part of a confirmation screen displayed in a pop-up window each time aprint instruction is issued.

To print an image on a sheet of paper, the information processingapparatus 1 converts three-dimensional information defining thethree-dimensional object into two-dimensional information. Thetwo-dimensional information herein may be, for example, an imageobtained by observing the surface of the three-dimensional object, or animage reflecting a change in an internal structure of thethree-dimensional object as illustrated in FIG. 24 or the like.

Note that the so-called printer may be, in addition to an apparatus thatprints a two-dimensional image on a recording medium such as a sheet ofpaper, a three-dimensional printer that forms a three-dimensional image.If the output destination is a three-dimensional printer, thethree-dimensional object is output as a three-dimensional object withoutbeing converted into three-dimensional information.

In the above-described exemplary embodiments, a mobile informationterminal (mobile display device) such as a tablet computer or asmartphone is assumed as the information processing apparatus 1.However, the above technique is applicable to any information terminalincluding a display (including projection function) and a sensor, suchas a clock, a toy like a game machine, a television receiver, aprojector, a head-mounted display, an image forming apparatus (so-calledprinter), an electronic white board, or a robot for communication.

Based on above, the invention provides a display control methodincluding following steps. Acquiring positional information, whichindicates a position where a mobile display device is being used, fromthe display device; acquiring first information, which is different fromthe positional information and is associated with the positionalinformation, from a device different from the display device; anddifferentiating a two-dimensional image, which represents an objectdefined three-dimensionally from one viewpoint direction and isdisplayed on the display device, according to the acquired firstinformation, and displaying the two-dimensional image as a moving image.An image of a virtual creature displayed on the display device, a formof which is imitated by the object, changes in association with at leastthe first information and the positional information. An image of avirtual creature different from the image of the virtual creature isadditionally displayed on the display device in association with achange of the first information and the positional information. In partof embodiments, the number of images of virtual creatures displayed onthe display device, forms of which are imitated by the object, isdifferent in association with the first information and the positionalinformation. In part of embodiments, the first information isinformation regarding weather and an abnormality of weather is notifiedthrough the information regarding weather, the two-dimensional image isdisplayed in a mode according to a content of the abnormality. Warninginformation is displayed on the display device when the abnormality ofweather is notified through the first information. In part ofembodiments, a magnitude of change in display of the two-dimensionalimage is changed according to the information regarding weather. In partof embodiments, the two-dimensional image increases on the displaydevice and then decreases according to the first information.

The foregoing description of the exemplary embodiments of the presentinvention has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit theinvention to the precise forms disclosed. Obviously, many modificationsand variations will be apparent to practitioners skilled in the art. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, therebyenabling others skilled in the art to understand the invention forvarious embodiments and with the various modifications as are suited tothe particular use contemplated. It is intended that the scope of theinvention be defined by the following claims and their equivalents.

What is claimed is:
 1. A display control method, comprising: acquiringpositional information, which indicates a position where a displaydevice is being used, from the display device, wherein the displaydevice is a mobile display device; acquiring first information, which isdifferent from the positional information and is associated with thepositional information, from a device different from the display device;and differentiating a two-dimensional image, which represents an objectdefined three-dimensionally from one viewpoint direction and isdisplayed on the display device, according to the acquired firstinformation, and displaying the two-dimensional image as a moving image.2. The display control method according to claim 1, wherein an image ofa virtual creature displayed on the display device, a form of which isimitated by the object, changes in association with at least the firstinformation and the positional information.
 3. The display controlmethod according to claim 2, wherein an image of a virtual creaturedifferent from the image of the virtual creature is additionallydisplayed on the display device in association with a change of thefirst information and the positional information.
 4. The display controlmethod according to claim 2, wherein the number of images of virtualcreatures displayed on the display device, forms of which are imitatedby the object, is different in association with the first informationand the positional information.
 5. The display control method accordingto claim 1, wherein when the first information is information regardingweather and an abnormality of weather is notified through theinformation regarding weather, the two-dimensional image is displayed ina mode according to a content of the abnormality.
 6. The display controlmethod according to claim 5, wherein when the abnormality of weather isnotified through the first information, warning information is displayedon the display device.
 7. The display control method according to claim1, wherein when the first information is information regarding weather,a magnitude of change in display of the two-dimensional image is changedaccording to the information regarding weather.
 8. The display controlmethod according to claim 1, wherein the two-dimensional image increaseson the display device and then decreases according to the firstinformation.
 9. An information processing apparatus, comprising adisplay controller which: acquires positional information, whichindicates a position where a mobile display device is being used, fromthe display device; acquires first information, which is different fromthe positional information and is associated with the positionalinformation, from a device different from the display device; anddifferentiates a two-dimensional image, which represents an objectdefined three-dimensionally from one viewpoint direction and isdisplayed on the display device, according to the acquired firstinformation, and displays the two-dimensional image as a moving image.10. The information processing apparatus according to claim 9, whereinan image of a virtual creature displayed on the display device, a formof which is imitated by the object, changes in association with at leastthe first information and the positional information.
 11. Theinformation processing apparatus according to claim 10, wherein an imageof a virtual creature different from the image of the virtual creatureis additionally displayed on the display device in association with achange of the first information and the positional information.
 12. Theinformation processing apparatus according to claim 10, wherein thenumber of images of virtual creatures displayed on the display device,forms of which are imitated by the object, is different in associationwith the first information and the positional information.
 13. Theinformation processing apparatus according to claim 9, wherein when thefirst information is information regarding weather and an abnormality ofweather is notified through the information regarding weather, thedisplay controller displays the two-dimensional image in a modeaccording to a content of the abnormality.
 14. The informationprocessing apparatus according to claim 13, wherein when the abnormalityof weather is notified through the first information, the displaycontroller displays warning information on the display device.
 15. Theinformation processing apparatus according to claim 9, wherein when thefirst information is information regarding weather, the displaycontroller changes a magnitude of change in display of thetwo-dimensional image according to the information regarding weather.16. The information processing apparatus according to claim 9, whereinthe display controller increases and then decreases the two-dimensionalimage on the display device according to the first information.
 17. Anon-transitory computer readable medium storing a program, enabling acomputer to: acquire positional information, which indicates a positionwhere a mobile display device is being used, from the display device;acquire first information, which is different from the positionalinformation and is associated with the positional information, from adevice different from the display device; and differentiate atwo-dimensional image, which represents an object definedthree-dimensionally from one viewpoint direction and is displayed on thedisplay device, according to the acquired first information, and displaythe two-dimensional image as a moving image.